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DETAILED ACTION 

Response to Amendment 

This action is Final. Claims 1-24 are currently pending. Applicant's response 
received on 2/19/2008 is fully considered herein and is not persuasive. 

Claim Rejections - 35 USC § 102 

1. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

2. Claims 1-3, 11, 14, 15 and 18 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Belaid et al. (Hereinafter "Belaid": Y. Belaid, A. Belaid and E. Turolla, 
"Item Searching in Forms: Application to French Tax Form", 1995, IEEE Proceedings of 
the third international conference on Document analysis and recognition, Montreal, 
Que., Canada, pages 744-747 (August 14-16, 1995)). 

Regarding claims 1, 14 and 18, Belaid teaches a method, equivalent 
computer program product embodied on at least one computer readable medium, and 
equivalent device, of segmenting a composite image of pixels into a number of fields 
corresponding to lay-out elements of the image, the pixels having a value representing 
the intensity and/or color of a picture element, the method comprising: constructing 
separating elements corresponding to rectangular areas of adjacent pixels of the image 
having a background property indicative of a background of the image (page 745, Fig. 
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2, i.e. real segments of the Hough line, corresponding black lines 
representing background elements of the image); and constructing a graph 
representing the lay-out elements of the image by defining vertices of the graph on the 
basis of intersections of separating elements that are substantially oriented in 
predetermined separation directions (page 745, § 3, H 1), defining edges of the graph 
between the vertices corresponding to the separating elements (page 745, § 3, H 1, 
i.e. arc link), and defining field separators corresponding to the edges of the graph 
(page 744, § 2, H 1, i.e. cell boundaries). 

Summary of Applicant's Remarks: Regarding independent claims 1, 14, 18 
and their respective dependent claims, Belaid et al. does not disclose the feature of 
"constructing separating elements corresponding to rectangular areas of adjacent pixels 
of the image having a background property indicative of a background of the image". 
Belaid et al. operates on existing black lines, whereas the present invention operates on 
background areas. Black lines do not represent background elements of the image. 

Examiner's Response: Belaid et al. teaches black areas (page 745, left 
column, lines 14-17, i.e. thick lines, which inherently have an area). Furthermore Belaid 
et al. discloses that black is a background property of the image (page 744, Section 2, 
paragraph 1, lines 3-5). Therefore, Belaid et al. does disclose the feature of 
"constructing separating elements corresponding to rectangular areas of adjacent pixels 
of the image having a background property indicative of a background of the image". 
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Regarding claims 2 and 15, Belaid teaches the method and equivalent 
computer program product as claimed in claim 1, wherein the step of defining vertices 
comprises: constructing subsets of the separating elements that are substantially 
oriented in the predetermined separation directions (page 745, Fig. 2, i.e. horizontal 
and vertical), and determining the intersections between pairs of separating elements 
from the subsets (page 745, § 3, U 2). 

Regarding claim 3, Belaid teaches the method as claimed in claim 2, wherein 
the step of determining the intersections comprises: determining an area of overlap of 
the separating elements from the subsets (page 745, § 3, H 2, lines 1-3, i.e. the 
intersection of the vertical and horizontal lines), and locating the vertex at the 
center of the area of overlap (i.e. the intersection of the corresponding Hough 
lines which are centered in their respective vertical and horizontal strips, 
page 745, left column, lines 18-22). 

Regarding claim 11, Belaid teaches the method as claimed in claim 1, wherein 
the step of constructing the separating elements comprises: processing the image in 
two orthogonal separation directions (page 745 § 2, H 2, lines 7-9, i.e. vertical and 
horizontal directions). 
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Claim Rejections - 35 USC §103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented 
and the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Belaid as 
applied to claim 1 above, and further in view of Abdel-Mottaleb et al. (Hereinafter 
"Abdel-Mottaleb": US 6263113). 

Belaid teaches the method as claimed in claim 1. 

Belaid does not teach the limitation wherein the graph constructing step 
comprises: assigning a weight to the edges indicating an Euclidean distance between 
the vertices. 

However Abdel-Mottaleb discloses a graph constructing step comprising: 
assigning a weight to the edges indicating an Euclidean distance between the vertices 
(column 5, lines 31-33). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Abdel-Mottaleb's step to Belaid's graph 
construction, the motivation being to detect form cells in a complex background 
(Abdel-Mottaleb: column 1, lines 48-50). 
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5. Claims 5, 10, 17 and 20 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Belaid as applied to claim 1 above, and further in view of 
Antonacopoulos et al. (Hereinafter "Antonacopoulos": A. Antonacopoulos and R. T. 
Ritchings, "Flexible Page Segmentation Using the Background", 1994, IEEE Proceedings 
of Computer Vision and Image Processing, Vol. 2, pp. 339-344). 

Regarding claims 5, 17 and 20, Belaid teaches the method, equivalent 
computer program product, and equivalent device, as claimed in claim 1. 

Belaid does not teach the limitation to the method as further comprising: 

constructing a set of maximal rectangles, a maximal rectangle being a 
rectangular part of the image in one of the separation directions, that has the maximum 
possible area without including a pixel not having the background property indicative of 
a background of the image; and 

constructing the separating elements in a cleaning step wherein at least one pair 
of overlapping maximal rectangles in the set is replaced by an informative rectangle 
that is a rectangular part of an area combining the areas of the pair, the rectangular 
part having the maximum possible length in the relevant separation direction. 

However Antonacopoulos discloses an image segmentation method comprising 
the steps of constructing a set of maximal rectangles, a maximal rectangle being a 
rectangular part of the image in one of the separation directions, that has the maximum 
possible area without including a pixel not having the background property indicative of 
a background of the image (page 341, § 2.2, U 1); and constructing the separating 
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elements in a cleaning step wherein at least one pair of overlapping maximal rectangles 
in the set is replaced by an informative rectangle that is a rectangular part of an area 
combining the areas of the pair, the rectangular part having the maximum possible 
length in the relevant separation direction (page 341, Figure 4, white tile A). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Antonacopoulos' maximal rectangle steps 
to Belaid's image segmentation method, the motivation being to identify white space 
separators (Belaid: Page 746, § 5, HI, lines 9-11). 

Regarding claim 10, Belaid in combination with Antonacopoulos teaches the 
method as claimed in claim 5, wherein the step of constructing the maximal rectangles 
comprises: determining a list of maximal runs, a maximal run being a straight line of 
pixels having the background property, the line having the maximum possible length 
without including a pixel not having the background property, taking a specific maximal 
run from the list as rectangle, testing the rectangle if extension is possible by 
determining for a next maximal run if the next maximal run comprises pixels adjacent 
to pixels of the rectangle in a width direction, if the extension is possible, extending the 
rectangle by constructing a new rectangle having the maximum area including pixels of 
the rectangle and the next maximal run, if no extension is possible, adding the 
rectangle to the set of maximal rectangles, and eliminating from the list any maximal 
run that is completely contained in the new rectangle (Antonacopoulos: page 341, § 
2.2 H 2-4, i.e. the Antonacopoulos algorithm is substantially the same, and 
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produces the same result as the recited algorithm. While Antonacopoulos 
does not explicitly disclose the use of a list in his algorithm, the use of lists to 
facilitate adding, sorting and removing items was well know in the art at the 
time of the invention and would therefore have been obvious to try). 

6. Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over Belaid in 
combination with Antonacopoulos as applied to claim 5 above, and further in view of 
Katsuyama (US 6226402). 

Belaid in combination with Antonacopoulos teaches the method as claimed in 
claim 5. 

Belaid in combination with Antonacopoulos does not teach the limitation wherein 
the cleaning step further comprises at least one of the following: deleting a maximal 
rectangle having a length below a predefined value, deleting a maximal rectangle 
having a width below a predefined value, and deleting a maximal rectangle having an 
aspect ratio below a predefined value, the aspect ratio being a longer side length 
divided by a shorter side length. 

However Katsuyama discloses a cleaning step further comprising at least one of 
the following: deleting a maximal rectangle having a length below a predefined value, 
deleting a maximal rectangle having a width below a predefined value, and deleting a 
maximal rectangle having an aspect ratio below a predefined value, the aspect ratio 
being a longer side length divided by a shorter side length (See Fig. 16, for the 
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horizontal case, and note that the same process can be applied to the vertical 
case (column 16, lines 4 and 5)). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Katsuyama's deleting method to the 
cleaning step of Belaid combined with Antonacopoulos, the motivation being to extract 
ruled line equivalent separators (Katsuyama: column 2, lines 62-65). 
7. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Belaid in 
combination with Antonacopoulos as applied to claim 5 above, and further in view of 
McLeod et al. (Hereinafter "McLeod": US 5778092). 

Belaid in combination with Antonacopoulos teaches the method as claimed in 
claim 5. 

Belaid in combination with Antonacopoulos does not teach the limitation wherein, 
prior to the step of constructing the maximal rectangles, the image is preprocessed by 
at least one of the following: removing noise by adapting the value of isolated deviant 
pixels to an average value of pixels in the neighborhood, halftoning by transforming the 
pixels to either white or black, and reducing the number of pixels by downsampling. 

However McLeod discloses a preprocessing step wherein the image is 
preprocessed by at least one of the following: removing noise by adapting the value of 
isolated deviant pixels to an average value of pixels in the neighborhood (column 10, 
line 25), halftoning by transforming the pixels to either white or black (column 9, 
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lines 57-60, i.e. thresholding), and reducing the number of pixels by downsampling 
(column 2, lines 6-11, i.e. reduced resolution). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied McLeod's preprocessing method prior to 
the step of constructing the maximal rectangles of Belaid combined with 
Antonacopoulos, the motivation being to enhance separation of text and images from 
page background (McLeod: column 10, lines 43-45). 

8. Claims 8 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Belaid in combination with Antonacopoulos as applied to claim 5 above, and further in 
view of Huang et al. (Hereinafter "Huang": US 5416849). 

Regarding claim 8, Belaid in combination with Antonacopoulos teaches the 
method as claimed in claim 5. 

Belaid in combination with Antonacopoulos does not teach the limitation wherein, 
prior to the step of constructing the maximal rectangles, the image is filtered by 
detecting foreground separator elements that are objects in the foreground of the 
image having a pattern of pixel values deviating from said background property, and by 
replacing pixels of the detected foreground separators by pixels having the background 
property. 

However, Huang discloses a filtering step wherein the image is filtered by 
detecting foreground separator elements that are objects in the foreground of the 
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image having a pattern of pixel values deviating from said background property 
(column 14, lines 21-23, i.e. long line segments), and by replacing pixels of the 
detected foreground separators by pixels having the background property (see Figs. 
2C and 2D). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Huang's filtering method prior to the step 
of constructing the maximal rectangles of Belaid combined with Antonacopoulos, the 
motivation being to improve field extraction by removing extraneous lines (Huang: 
column 7, lines 49-53). 

Regarding claim 9, Belaid in combination with Antonacopoulos and further in 
view of Huang, teaches the method as claimed in claim 8, wherein the foreground 
separator elements include black lines (Huang: column 14, line 22), dashed lines, or 
dotted lines (Huang: column 19, BLOB, i.e. dashed and dotted lines are 
encompassed by the BLOB primitive). 

9. Claims 12, 13, 16 and 21 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Belaid as applied to claim 1 above, and further in view of Sakai et al. 
(Hereinafter "Sakai": US 5949555). 

Regarding claim 12, Belaid teaches the method as claimed in claim 1, wherein 
the step of constructing the separating elements comprises: detecting graphical 
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elements that are objects in the foreground of the image having a pattern of pixel 
values deviating from said background property (page 744 § 2, i.e. lines). 

Belaid does not teach the limitation wherein the separating elements are 
constructed around the graphical elements. 

However, Sakai discloses a method wherein the separating elements are 
constructed around the graphical elements (Figs. 30-32). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Sakai's separation method to Belaid's 
application, the motivation being to improve field extraction by precise classification of 
field attributes (Sakai: column 1, lines 47 and 48). 

Regarding claims 13, 16 and 21, Belaid teaches the method, equivalent device, 
and equivalent computer program product as claimed in claim 1. 

Belaid does not teach the limitation wherein at least one of the fields is classified 
as text field, a reading order is detected in the text field, and foreground components 
are joined to text lines in the text field in a direction corresponding to said reading 
order. 

However, Sakai discloses a method, equivalent computer program product, and 
equivalent device wherein at least one of the fields is classified as text field (column 
20, lines 17-19), a reading order is detected in the text field (column 20, lines 29- 
36, i.e. vertical or horizontal direction), and foreground components are joined to 
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text lines in the text field in a direction corresponding to said reading order (e.g. see 
Figs. 30A, Title Portion). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Sakai's separation method to Belaid's 
application, the motivation being to improve field extraction by precise classification of 
field attributes (Sakai: column 1, lines 47 and 48). 

10. Claim 19 is rejected under 35 U.S.C. 103(a) as being unpatentable over as 
applied to claim 1 above, and further in view of Mahoney et al. (Hereinafter "Mahoney": 
US 6470095). 

Belaid teaches the device as claimed in claim 18. 

Belaid does not disclose the device further comprising: a display unit for 
displaying fields of the image after segmenting. 

However Mahoney discloses a display unit for displaying fields of the image after 
segmenting (Fig. 1, Display Monitor, and Fig. 21). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have utilized Mahoney's display device, the motivation 
being to provide user feedback (Mahoney: column 4, lines 1-3). 
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11. Claims 22-24 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Belaid as applied to claim 1 above, and further in view of Antonacopoulos. 

Regarding claims 22-24, Belaid teaches the method, equivalent computer 
program product, and equivalent device, as claimed in claim 1. 

Belaid does not teach the limitation to the method wherein the rectangular areas 
are white areas. 

However Antonacopoulos discloses an image segmentation method using 
maximal rectangles wherein the rectangular areas are white (page 340, section 2, 
paragraph 2, i.e. white spaces of varying sizes). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Antonacopoulos' maximal rectangle steps 
to Belaid's image segmentation method, the motivation being to identify white space 
separators (Belaid: Page 746, § 5, ^1, lines 9-11, i.e. Belaid clearly discloses 
the need for a method to search for white bands and Antonacopoulos 
discloses such a method). 

Conclusion 

12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of 
time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
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TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire 
later than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to CLIFTON G. DALEY whose telephone number is 571- 
270-3144. The examiner can normally be reached on Monday - Friday 7:30am - 
4:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Samir Ahmed can be reached on 571-272-7413. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO 
Customer Service Representative or access to the automated information system, call 
800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Samir Ahmed 
Examiner 
Art Unit 2624 

CGD 

6/9/2008 



/Samir A. Ahmed/ 

Supervisory Patent Examiner, Art Unit 2624 



